Augmented reality feedback of inventory for an appliance

ABSTRACT

A food storage system is disclosed. The system comprises an appliance having a cabinet defining an interior, the interior comprising at least one of a shelf, compartment, or drawer configured to store at least one food item, a vision system comprising at least one imager configured to record image data related to the at least one food item stored within the interior, and a controller in communication with the vision system. The controller can be operable to generate an augmented reality representation of the interior.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.16/237,066, now U.S. Patent Publication No. 2020/0211285, filed Dec. 31,2018, and entitled “AUGMENTED REALITY FEEDBACK OF INVENTORY FOR ANAPPLIANCE,” the entire disclosure of which is incorporated herein byreference in its entirety.

TECHNOLOGICAL FIELD

The present device generally relates to a food storage system and, moreparticularly, relates to a food storage system for an appliance.

SUMMARY

In at least one aspect, a food storage system is disclosed. The systemcan comprise an appliance having a cabinet defining an interior, theinterior comprising at least one of a shelf, compartment, or drawerconfigured to store at least one food item, a vision system comprisingat least one imager configured to record image data related to the atleast one food item stored within the interior, and a controller incommunication with the vision system. The controller can be operable todetermine an identity of the least one food item, determine a storageconfiguration of the interior, generate an augmented realityrepresentation of the interior based on the determined identity of theat least one food item and the storage configuration of the interior,and display the augmented reality representation. The augmented realityrepresentation can comprise a digital rendering of the storageconfiguration and a digital food item representation corresponding tothe at least one food item arranged within the digital rendering.

In at least another aspect, a food storage system is disclosed. Thesystem can comprise an appliance having a cabinet defining an interior,the interior comprising at least one of a shelf, compartment, or drawerconfigured to store at least one food item, a vision system comprisingat least one imager configured to record image data related to the atleast one food item stored within the interior, a lighting systemcomprising at least one light source configured to project a pattern oflight, and a controller in communication with the vision system. Thecontroller can be operable to determine an identity and a location ofthe at least one food item, analyze a query from a user regarding the atleast one food item, and control the lighting system to project apattern of light onto the at least one food item in response to thequery. The pattern of light can provide feedback to the user indicatingat least one of a location or a food item property of the at least onefood item.

In at least another aspect, a food storage system is disclosed. Thesystem can comprise an appliance having a cabinet defining an interior,the interior comprising at least one of a shelf, compartment, or drawerconfigured to store at least one food item, a vision system comprisingat least one imager configured to record image data related to the atleast one food item stored within the interior, and a controller incommunication with the vision system. The controller can be operable todetect at least one food item property for the at least one food item,generate an augmented reality representation of the interior based on auser selection of a food item property, and display the augmentedreality representation. The augmented reality representation cancomprise at least one digital food item overlay corresponding to thefood item property.

These and other features, advantages, and objects of the present devicewill be further understood and appreciated by those skilled in the artupon studying the following specification, claims, and appendeddrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings:

FIG. 1 is a front view of an appliance interior comprising a foodstorage system according to various aspects described herein;

FIG. 2 is a schematic view of a portion an interior of an applianceaccording to various aspects described herein;

FIG. 3 is a schematic view of an augmented reality representation of anappliance interior according to various aspects described herein;

FIG. 4 is a schematic view of an augmented reality representation of anappliance interior according to various aspects described herein;

FIG. 5 is a schematic view of an augmented reality representation of anappliance interior according to various aspects described herein;

FIG. 6 is a schematic view of an augmented reality representation of anappliance interior according to various aspects described herein;

FIG. 7A is a block diagram of a food storage system according to variousaspects described herein;

FIG. 7B is a block diagram of a food storage system according to variousaspects described herein; and

FIG. 8 is a flow chart demonstrating a process for a user querying thefood storage system according to various aspects described herein.

DETAILED DESCRIPTION OF EMBODIMENTS

For purposes of description herein, the terms “upper,” “lower,” “right,”“left,” “rear,” “front,” “vertical,” “horizontal,” and derivativesthereof shall relate to the device as oriented in FIG. 1. However, it isto be understood that the device may assume various alternativeorientations and step sequences, except where expressly specified to thecontrary. It is also to be understood that the specific devices andprocesses illustrated in the attached drawings, and described in thefollowing specification are simply exemplary embodiments of theinventive concepts defined in the appended claims. Hence, specificdimensions and other physical characteristics relating to theembodiments disclosed herein are not to be considered as limiting,unless the claims expressly state otherwise.

Referring to FIG. 1, a storage system 10 for an appliance 12, such as arefrigerator, is shown. The appliance 12 can include a cabinet 14defining an interior 16. The cabinet 14 may include a display screen 36,which can be a LCD or any suitable display. The display screen 36 may bein the form of a user interface. The interior 16 may include a pluralityof holders defining a storage configuration to store at least one item24, which may include a food item 24. The holders may in include atleast one of a shelf 18, compartment 20, or drawer 22 and anycombination thereof. While the interior 16 is shown to include acombination of shelves 18, compartments 20, and drawers 22, it is notlimited to these storage configurations and may include any suitablestructure for storing a food item 24. Furthermore, the interior 16 mayinclude a lighting system including at least one light source 28. The atleast one light source 28 may be configured to provide selectivelighting, such as a pattern of light 30, and may be in the form of anysuitable light source 28, such as a micro projector, a laser, and thelike.

The storage system 10 may be configured to recognize and track a statusof the item 24 stored within the appliance 12. The specific status ofthe item 24 tracked or updated by the system 10 may vary depending onthe nature of the appliance 12. Accordingly, the disclosure may providefor a storage system 10 that may be utilized to recognize the status ofan item, inventory of an item, and/or various processing states togather and track various information as discussed herein.

For example, if the appliance 12 corresponds to a refrigerator, asillustrated in FIG. 1, the system 10 may be configured to track aninventory of the food item 24 as it is added or removed from theinterior 16. In order to detect and identify the food item 24, thestorage system 10 may comprise a vision system including at least oneimager 26. In some embodiments, the at least one imager 26 maycorrespond to a plurality of imagers 26. The at least one imager 26 maybe configured to capture image data in the interior 16 or in an accessregion external to the appliance 12. The storage system 10 may beoperable to detect the food item 24 in connection with at least one partof a person's body (e.g. a hand) in connection with the food item 24entering and/or exiting the interior 16 via the access region. In thisway, the system 10 may be operable to track the status of the food item24 by detecting the part of the person's body as it enters and exits theaccess region. A system operable to track the status of a food item bydetecting part of a person's body as it enters and exits the accessregion may include the system disclosed in U.S. Pat. No. 10,157,308,issued Dec. 18, 2018, entitled “Interaction Recognition and AnalysisSystem,” which is herein incorporated by reference in full. U.S. Pat.No. 10,157,308 discloses an object tracking system configured to trackan inventory of the object as it is added or removed from an operatingvolume of a refrigerator.

The vision system may further include one or more infrared (IR) cameras,or sensors. An infrared camera may be incorporated into the system 10 inorder to detect a temperature of a food item 24 or the temperature ofvarious locations within the interior 16 of the appliance 12.

The item 24 may generally correspond to any form of object that may bestored, processed, disposed of, or otherwise utilized in connection withan appliance 12. As previously discussed, the item 24 may correspond toa food item 24, or product, that may be stored in the interior 16 of therefrigerator 12. The system 10 may process image data captured by the atleast one imager 26 in order to identify a product type and proportionor quantity by utilizing various imaging processing techniques. With theproduct type and quantity identified for the food item 24, the system 10may update an inventory status of the product type and quantity of thefood item 24 in a memory or inventory database. Though discussed inreference to an inventory status, the system 10 may be configured todetect various forms of information in reference to the food item 24,which may include, but are not limited to, a depletion or usage, alocation, a quality status (e.g. the presence of mold), a color orconsistency, and/or various additional information that may be derivedfrom the image data.

In some embodiments, the system 10 may be operable to track variousforms of information regarding the status and characteristics of thefood item 24. As discussed herein, such information may be inferred bythe system 10 based on a process completed by the appliance 12 and/or aduration of time between a first detection of the food item 24 and asecond detection of the food item 24 (e.g. a time between removal andplacement of the object in the operating volume, or interior 16). Suchinformation may include clock and calendar data for inventory andquality tracking of the food item 24. A status or characteristic of thefood item 24 may also be inferred by monitoring of depletion, or filllevel, of the food item 24. Additionally, the status of the food item 24may be indicated based on actions taken on the object including, but notlimited to, wrapping the food item 24, opening a package containing theobject, covering a surface or container of the object, etc.

Based on the image data, the system 10 may identify the food item 24.Details regarding image processing techniques that may be utilized toidentify various food items 24 and corresponding states of such fooditems 24 are further discussed in reference to FIGS. 7A and 7B. Theidentification of the food item 24 may include an indication of a typeor category to which the food item 24 belongs and a proportion orquantity of the food item 24. The system 10 may further identifyadditional information about the food item 24, including, but notlimited to, a color, a texture, a storage data, a location, and variousadditional information. In some embodiments, the system 10 may classifythe food item 24 into classes and subclasses. For example, in the casewhere the food item 24 is a food product, the system 10 can determine aclass of the food item 24 as a beverage, a protein, a fruit, avegetable, a packaged industrial food, a dressing, and the like. Asubclass of a beverage may include a can, a bottle, a box, and the like,while a subclass of a protein may include a type, such as chicken, beef,fish, a state, such as frozen or fresh, and/or a size.

The system 10 may be operable to detect and update the status of thefood item 24 based on a variety of properties and/or characteristicsthat may be identified in the image data received from the imager 26.For example, as previously discussed, an identification of the food item24 within the interior 16, or in connection with the hand being insertedor removed from the operating volume 20 of the refrigerator 12 may beutilized by the system 10 to update an inventory of the food item 24.The system 10 may track a variety of properties and/or characteristicscorresponding to the food item 24 based on the image data captured bythe imager 26.

For example, the system 10 may detect an identity, color, surfacetexture, quantity, fill level, proportion, or any form of informationthat may be visually and/or thermally detected by the system 10 based onthe image data received from the imager 26. The system 10 may drawvarious inferences regarding the status of the food item 24 based on theimage data. In some scenarios, the system 10 may also infer informationabout the food item 24 based on a duration of time that the object isdetected as being removed from the interior 16.

Accordingly, a controller of the system 10 may be in communication withan appliance control system to determine a food item property for thefood item 24 based on one or more properties and/or characteristicsidentified from the image data. The controller of the system 10 may beconfigured to identify the food item 24 by utilizing an objectidentification module and determine the appropriate food item property.Furthermore, based on identification of the food item 24 in the imagedata captured by the imager 26, the system 10 may update a usage orinventory of the food item 24 as being consumed or depleted. A food itemproperty can be in the form of a fill level, an expiration date, afavorite, a recipe, a quantity, a brand, a condition, a placement, aname, a type, and the like, relating to the food item 24. An objectdetection module may detect the location food item 24.

FIG. 2 illustrates a portion of the interior 16 of the appliance 12. Thesystem 10 may be configured to determine a storage configuration 32 ofthe interior 16. The storage configuration 32 can include the relativelocation of different food items 24 to one another, which can be trackedto create a full view of how food items 24 are stored in the appliance12, or refrigerator. Additionally, the system 10 can determine thelocation, identity, and size of the holders, such as the shelves 18,compartments 20, or drawers 22, to further create the full view of thestorage configuration 32. In some examples, the storage configuration 32of the appliance 12 can be cluttered with many food items 24, such thata user viewing the interior 16 in real-time, or via a display (e.g. amobile display) in communication with the vision system, may considerthe interior 16 to be chaotic and messy.

In some examples, the system 10 can track and monitor the location,identity, and size of the food item 24 holders by way of a 3-Dimensionalcoordinate system within the interior 16. The system 10 may determine aunique set of coordinates for each shelf 18, compartment 20, drawer 22,etc. located within the interior 16. The set of coordinates may definethe storage configuration 32. Furthermore, the coordinates may be storedin a memory, or library, such that the system 10 can refer to thecoordinates as needed.

The system 10 can be configured to provide an augmented reality view ofthe interior 16 in order to “clean-up”, or organize, the view for theuser on a display, such as a mobile device display or the appliance userinterface, or display 36 (FIG. 1), but is not limited to such. Anaugmented reality view can be in the form of an interactive experienceof the real-word interior 16 of the appliance 12 where the storageconfiguration 32 and the food items 24 are “augmented” or “enhanced” bydata provided by the system 10. Turning to FIG. 3, a mobile device 41includes a display 42 on which a user can view an interior rendering 116of the appliance 12 interior 16. In this way, the interior rendering 116is organized and more easily viewable than the interior 16 shown in FIG.2. Thus, a user may avoid “hunting” for food items 24 within theinterior 16.

The interior rendering 116 can be in the form of a digital rendering ofthe storage configuration 32 (FIG. 2) which may include at least onedigital food item representation 124. The digital food itemrepresentation 124 may be in the form of a digital graphic, orgraphical, image that corresponds to a food item 24. As shown in FIG. 3,the digital food item representation 124 corresponds to an apple storedon a digital representation of a shelf 118. The interior rendering 116may further include digital food item representations of multiple fooditems, such as a carton of a gallon of milk 130, a lemon 132, and acarton of eggs 134, as well as a digital representation of the storageconfiguration 32 including digital representations of the shelves 118,compartments 120, and drawers 122, including size and relativepositions. Digital food item representations may include default imagesof an identified food item 24, or images specific to a food item brand.Additionally, a digital food item representation 124 may include asingle digital image for multiple identified food items 24. For example,if the system 10 identifies a total of six yogurts, the interiorrendering 116 may include a single digital image paired with a number toindicate the number of yogurts (6).

The system 10 may be configured to provide the interior rendering 116 inmore than one layer such that the user may view the contents of theinterior 16 over more than one view. Viewing the contents of theinterior 16 in layers may provide feedback that is easily viewable bythe user. In some examples, the user may view a layer that correspondswith a selection of a food item property. The food item property may bedetermined by the object identification and detection modules based onone or more properties, characteristics, and location of a food item 24.

Turning to FIG. 4, in some examples, a user can select a food itemproperty that corresponds to expiration dates of food items 24. Theinterior rendering 116 can include at least one digital food itemoverlay 138 that corresponds to the selected food item property. Thus,in the event that a user selects expiration dates as a food itemproperty for viewing, the digital food item overlay 138 may be in theform of a digital tag 138 including the expiration date of each fooditem 24, which may also be represented by the digital food itemrepresentation 130. The expiration date of a food item 24 may bedetermined by any suitable method, such as a user input, or identifiedby the vision system.

In another example, FIG. 5 illustrates the interior rendering 116corresponding to a layer generated by a user selection of a fill levelfood item property. One or more digital fill level icons 136 can bedisplayed within the interior rendering 116 to indicate the fill levelof a corresponding food item 24. The fill level icon 136 can be in theform of any digital image, or overlay, suitable to indicate anapproximate fill level. For example, the fill level icon 136 may bepositioned adjacent the gallon of milk, represented by the digital fooditem representation 130, and may include a level providing anapproximate amount of milk remaining in the gallon container locatedwithin the interior 16 of the appliance 12. The fill level, or usage, ofa food item 24 may be determined by any suitable method, such as a userinput, or identified by the vision system.

In yet another example, FIG. 6 illustrates the interior rendering 116corresponding to a layer generated from a user selection of a userfavorite's food item property. One or more digital favorite overlays 140can be displayed within the interior rendering 116 to indicate that aspecific food item 24 is a user favorite item. The digital favoriteoverlay 140 can be in the form of any digital image, overlay, or tag,suitable to indicate that a food item 24 is a favorite. For example, thedigital favorite overlay 140 may be positioned adjacent a container ofice cream determined to be located within a freezer drawer indicated bythe digital representation of the drawer 122, and may include a tagdisplaying a heart icon for indicating that a user's favorite item, suchas ice cream, is available within the interior 16 of the appliance 12.The user favorite designation of a food item 24 may be determined by anysuitable method, such as a user input, or identified by the visionsystem by user habits monitoring.

While food item properties selected by the user for augmented realityviewing generated by the system 10 have been described in relation toexpiration dates, fill levels, and favorites corresponding to variousfood items 24, the system 10 can generate an interior representation 116corresponding to food items 24 according to a variety of determinedcharacteristics of the food item 24. In further examples, the system 10can take into account the quality of the food item 24, recipes,quantities, etc. It is also possible that the user selects more than onefood item property such that multiple food item properties arerepresented in a single layer, or interior rendering 116.

Referring now to FIGS. 7A and 7B, block diagrams of exemplaryembodiments of the storage system 10 are shown. The system 10 maycomprise a controller 72 which may be integrated with or incommunication with an appliance controller 74. The controller 72 maycomprise one or more processors and/or memory configured to complete thevarious processing steps and control procedures discussed herein. In anexemplary embodiment, the controller 72 may comprise an image processor76 in communication with the at least one imager 26. The image processor76 may correspond to one or more microprocessors, discrete logiccontrollers or a combination of analog and/or digital circuitry. Theimage processor 76 may comprise a memory 78 and a communication circuit80.

The memory 78 may correspond to various forms of tangible ornon-transitory machine-readable or computer-readable media. For example,the memory 78 may correspond to flash memory, random access memory(RAM), erasable programmable read only memory (EPROM), or othermachine-readable media. The image processor 76 may store various imageprocessing algorithms, control instructions, and additional informationin the memory 78 to complete one or more of the processing tasksdescribed herein.

The communication circuit 80 may correspond to a wired or wirelesscommunication device through which the controller 72 may communicatewith and/or access various forms of information. For example, thecontroller 72 may access or update an inventory or grocery database 82via the communication circuit 80. The inventory or grocery database 82may correspond to a remote or cloud-based storage that may be accessibleby a variety of devices to access the inventory and/or grocery databaseinformation as described herein. Additionally, the communication circuit80 may further be in communication with an additional remote server ordatabase 84 from which controller software updates, object recognitioninformation, algorithms, object libraries, recipe libraries, and variousadditional information for use with the system 10 may be accessed.

Though discussed in reference to remote or cloud-based databases, thedatabases 82 and/or 84 may also correspond to local storage provided ina memory of the controller 72. Additionally, in some embodiments, aportion of each of the databases 82 and 84 may be stored in the localmemory while additional portions may be stored in the remote databases82 and 84 in communication with the controller 72 via the communicationcircuit 80. The communication circuit 80 may utilize various forms ofcommunication interfaces including, but not limited to, the followingwireless communication protocols: 3G, 4G, Wi-Fi®, Wi-Max®, CDMA, GSM,and/or any suitable wireless communication protocol.

As discussed herein, the system 10 may be operable to complete variousimage processing and identification steps corresponding to the food itemstored within the interior 16. Such detection, recognition, and/oridentification may be completed by the image processor 76 by processingthe image data received from the at least one imager 26. In order toprocess the image data, the image processor 76 may comprise a pluralityof processing modules configured to improve or simplify the image data,and/or determine an identity of the food item 24. In this way, the imageprocessor 76 may provide for the determination of the identity,quantity, and/or proportion of the food item 24.

In an exemplary embodiment, image processor 76 may comprise an imagesampler 92. The image sampler 92 may gather a sequence of image framesfrom the image data received from the at least one imager 26. Thesequence of image frames may be temporarily stored in a frame buffer 94.From the frame buffer 94, the image data from the image frames may beretrieved by a pre-processor 96. The pre-processor 96 may process thesequence of image frames from the image data to enhance the contrast,adjust a rotation, correct a boundary, and/or scale or crop the imagedata. The pre-processor 96 may accomplish such steps by processing theimage data with a variety of enhancement techniques, including, but notlimited to, histogram equalization, image segmentation, filtering, etc.

Enhanced image data may be communicated from the pre-processor 96 to theobject detection module 98. The object detection module 98 may processthe enhanced image data corresponding to the location of one or morefood items 24 as described herein. The image data may be communicated toan object identification module 100. The object identification module100 may process the image data to determine an identity, type, category,proportion, and/or quantity of the food item 24.

The object identification module 100 may utilize a variety of scanningmethods and/or algorithms to determine the identity of the food item 24.Such algorithms may include, but are not limited to, characterrecognition, feature extraction, clustering, pattern matching, neuralnetwork processing, optical character recognition (OCR), or variousother recognition techniques. In some embodiments, the objectidentification module 100 may further be in communication with an objectdatabase 102. The object database 102 may comprise a library of featuresand/or characteristics for comparison to the enhanced image data by theobject identification module 100 in order to determine the identity ofthe food item 24. In this way, the system 10 may be operable todetermine the identity of the food item 24 as discussed herein.

Although the image processor 76 is demonstrated as being an integralcomponent of the controller 72, the image processor 76 may correspond toa remote image processing system that may be in communication with thecontroller 72 via the communication circuit 80. In this way, the storagesystem 10 may communicate the image data to a remote image processor viathe communication circuit 80 in order to determine the identity of thefood item 24 with the object identification module 100. Upondetermination of the identity of the food item 24 from the image data,the remote image processor may provide an indication of the identity andany other relevant information to the controller 72 via thecommunication circuit 80. Accordingly, though the storage system 10 isdiscussed in reference to specific examples, the functions described inreference to the system 10 may be completed in a variety of ways withoutdeparting from the spirit of the disclosure.

In some embodiments, the at least one imager 26 may correspond to aplurality of imaging devices or a stereoscopic imaging device. In thisway, the image processor 76 may also be operable to determine a positionand proximity of the food item 24 based on depth data gathered by thestereoscopic imaging devices. Such positional information of the fooditem 24 may be beneficial in identifying or distinguishing a location ofthe food item 24. Accordingly, the storage system 10 may correspond to ascalable system operable to suit a variety of applications. Astereoscopic system may also be used to estimate volume of a food item24. In some embodiments, the at least one imager 26 may correspond to aninfrared imaging device, thermal imaging device, or a combination ofthermal and conventional imaging devices. The thermal imaging device maycorrespond to a focal plane array (FPA) utilizing microbolometers as FPAsensors. Accordingly, the image data captured by the system 10 maycomprise thermal image data and/or conventional image data in thevisible light spectrum. In some embodiments, the system 10 may beoperable to distinguish between a plurality of objects having a likevisual appearance in the image, but differing temperatures. For example,in some embodiments, the system 10 may be operable to track an inventoryof an object that is removed from the interior 16 at a cold temperatureand replaced in the operating volume at a warm temperature. Accordingly,by utilizing the thermal imaging data, the system 10 may be operable todistinguish additional status information for the food item 24.

Referring to FIGS. 1, 7A, and 7B, the controller 72 of the system 10 canbe configured to determine a query, or input, from the user regardingthe food item 24. A query from the user may be in the form of a gesture,a vocal communication, or any suitable haptic command. The controller 72can be configured to determine a response to the query, which may bebased on an identity or location of the food item 24. A response to thequery may include a voice message, selective lighting, etc. Examples ofselective lighting can include, but are not limited to, a pattern oflight 30 (FIG. 1) including a light spot on the food item 24, a lightcolor on the food item 24, or words or graphics generated by anarrangement of pixels, a pulsing light, dimming light, or lighting aspecific zone of the interior 16.

In some examples, a user may query the system 10 with a query thatincludes a voice message inquiring about an optimal storage location forthe food item 24, which may include, “Where do I store the butter”? Thecontroller 72 may analyze the query and determine the response forguiding the user for storage of the food item 24. For example, thesystem 10 may then provide a pattern of light 30 that can includeselective lighting of at least a portion of the appliance 12, such as acompartment 20, to provide an indication of the optimal storage locationfor the butter, or food item 24.

In another example, a user may query the system 10 with a query thatincludes a voice message inquiring about the location of a food item 24,which may include, “Where are my strawberries”? The controller 72 mayanalyze the query and determine the response for guiding the user to thelocation of the food item 24. For example, the system 10 may thenprovide a pattern of light 30 that can include a beam of light directedto the strawberries, or food item 24, to provide an indication of thelocation. The beam of light may include colors that correlate to adetermined food item property, such as a quality of the food. In someexamples, a red beam of light may indicate that the food item 24 hasexpired, or contains mold, while a green beam of light may indicate thatthe food item 24 is unexpired, or fresh.

In yet another example, a user may query the system 10 with a query thatincludes a voice message inquiring about available ingredients for arecipe, which may include, “What do I have for making a sandwich”? Thecontroller 72 may analyze the query and determine the response forproviding feedback to the user regarding multiple food items 24available for a recipe. For example, the system 10 may then provide apattern of light 30 that can include a beam of light directed to each ofthe food items 24, which may include bread, meat, vegetables, andcondiments stored within the interior 16 to provide an indication ofavailable food items 24 for making a sandwich.

While responses by the system 10 have been described in relation tostorage or recipes for the food item 24, the system 10 can determineresponses to a user query regarding a food item 24 to provide assistanceto the user using a variety of determined characteristics of the fooditem 24. In further examples, the system 10 can take into account thestatus of the food item 24, which can include a depletion/usage for thefood item 24 and determine a fill level.

In some examples, a response can include a heat map of the interior 16generated by the lighting system. The heat map can include lighting upthe interior 16 according to variations in temperature that maycorrespond to variations in colored light. In some examples, thevariations in colored light may indicate poor placement of food items24. For example, placing yogurt towards the rear, or back, of theinterior 16, can cause the yogurt to freeze. Poor placement may beindicated by red colored light, while optimal placement may be indicatedby green colored light. Thus, the heat map may include multiple lightprojections, or beams throughout the interior 16. Additionally,variations in colored light may indicate issues, or messes within theinterior 16. Messes may result from placing a large frozen mass, such asa frozen turkey, within the interior 16 where humidity may increase andlead to dripping. The heat map may indicate if dripping has occurred.

FIG. 8 illustrates a flow chart demonstrating a process 200 wherein auser queries the system 10 in regards to a food item 24. The process 200may begin at step 210 when the system 10 detects a user query regardinga food item. The user query can pertain to any food item property thatcan be detected or determined by the system 10 for the food item 24. Atstep 220, the controller 72 of the system 10 may analyze, or process,the query and then determine the appropriate response at step 230. Theappropriate response at step 230 can include feedback to the userregarding the food item 24, which may include any suitable audio orvisual techniques. At step 240, the controller 72 of the system 10 caninitiate the response. Initiating the response can include activatingthe light source 28 (FIG. 1). Following the initiating of the responseat step 240, the system 10 can provide the response and end the process200 at step 250.

Providing the response can include projecting the response onto the fooditem 24. Projecting the response onto the food item 24 can include anysuitable lighting technique to provide an indication, including, but notlimited to: words, icons, graphics, animations, light beams, diffusedlight, and coloration of light, and any combination thereof. Forexample, the projection 30 (FIG. 1) may include an expiration date. Theexpiration date can be in the form of colored pixels where the coloringmay provide an indication if the expiration date has passed or not, suchas green for not expired and red for expired.

While the method 200 is described as including steps 210-250, it ispossible for the method 200 to include additional or fewer steps.Additional steps can include any suitable step or process as describedherein. In some examples, the method 200 may include analyzing more thanone query from the user and determining more than one response.Furthermore, initiating the response at step 240 may occursimultaneously with providing the response at step 250.

It will be understood by one having ordinary skill in the art thatconstruction of the described device and other components is not limitedto any specific material. Other exemplary embodiments of the devicedisclosed herein may be formed from a wide variety of materials, unlessdescribed otherwise herein.

For purposes of this disclosure, the term “coupled” (in all of itsforms, couple, coupling, coupled, etc.) generally means the joining oftwo components (electrical or mechanical) directly or indirectly to oneanother. Such joining may be stationary in nature or movable in nature.Such joining may be achieved with the two components (electrical ormechanical) and any additional intermediate members being integrallyformed as a single unitary body with one another or with the twocomponents. Such joining may be permanent in nature or may be removableor releasable in nature unless otherwise stated.

It is also important to note that the construction and arrangement ofthe elements of the device as shown in the exemplary embodiments isillustrative only. Although only a few embodiments of the presentinnovations have been described in detail in this disclosure, thoseskilled in the art who review this disclosure will readily appreciatethat many modifications are possible (e.g., variations in sizes,dimensions, structures, shapes and proportions of the various elements,values of parameters, mounting arrangements, use of materials, colors,orientations, etc.) without materially departing from the novelteachings and advantages of the subject matter recited. For example,elements shown as integrally formed may be constructed of multiple partsor elements shown as multiple parts may be integrally formed, theoperation of the interfaces may be reversed or otherwise varied, thelength or width of the structures and/or members or connector or otherelements of the system may be varied, the nature or number of adjustmentpositions provided between the elements may be varied. It should benoted that the elements and/or assemblies of the system may beconstructed from any of a wide variety of materials that providesufficient strength or durability, in any of a wide variety of colors,textures, and combinations. Accordingly, all such modifications areintended to be included within the scope of the present innovations.Other substitutions, modifications, changes, and omissions may be madein the design, operating conditions, and arrangement of the desired andother exemplary embodiments without departing from the spirit of thepresent innovations.

It will be understood that any described processes or steps withindescribed processes may be combined with other disclosed processes orsteps to form structures within the scope of the present device. Theexemplary structures and processes disclosed herein are for illustrativepurposes and are not to be construed as limiting.

It is also to be understood that variations and modifications can bemade on the aforementioned structures and methods without departing fromthe concepts of the present device, and further it is to be understoodthat such concepts are intended to be covered by the following claimsunless these claims by their language expressly state otherwise.

The above description is considered that of the illustrated embodimentsonly. Modifications of the device will occur to those skilled in the artand to those who make or use the device. Therefore, it is understoodthat the embodiments shown in the drawings and described above is merelyfor illustrative purposes and not intended to limit the scope of thedevice, which is defined by the following claims as interpretedaccording to the principles of patent law, including the Doctrine ofEquivalents.

What is claimed is:
 1. An inventory management system for an appliance, the inventory management system comprising: a holder disposed in an interior of the appliance and configured to store a food item; an imaging device configured to capture an image of the interior; a display in communication with the imaging device configured to present image data corresponding to the image; and a controller in communication with the imager and the display, the controller configured to: determine an identity of the food item; determine a target location for the food item based on the identity of the food item and an environmental condition of the interior; and communicate an instruction to present the image data with an indication of the target location at the display.
 2. The inventory management system according to claim 1, wherein the image data includes an augmented reality representation including a photographic representation of the interior with a digital representation of the food item overlaying a digital representation of a position of the food item in the interior.
 3. The inventory management system according to claim 2, wherein the controller is further configured to generate the augmented reality representation based on the identity of the food item.
 4. The inventory management system according to claim 2, wherein the indication is a color indicator and the photographic representation of the interior includes the color indicator.
 5. The inventory management system according to claim 2, wherein the digital representation of the food item is presented as rendered graphics of the at least one food item.
 6. The inventory management system according to claim 1, wherein the controller is further configured to: determine an expiration date of the food item based on a position of the food item in the interior and a heat distribution of the interior; and communicate an instruction to present the image data with the expiration date at the display.
 7. The inventory management system according to claim 1, wherein the image data includes an overlay corresponding to a property of the food item.
 8. The inventory management system according to claim 7, wherein the property is a fill level and the overlay is a graphical image configured to indicate the fill level of the food item.
 9. The inventory management system according to claim 7, wherein the property is an expiration date and the overlay is a graphical image indicating the expiration date of the food item.
 10. The inventory management system according to claim 7, wherein the property is a favorite and the overlay is a graphical image indicating that the food item is a favorite. 11: An inventory management system for an appliance, the inventory management system comprising: at least one holder disposed in an interior of the appliance and configured to store a plurality of food items; an imaging device configured to capture an image of the interior; a display in communication with the imaging device configured to present image data corresponding to the image; and a controller in communication with the imager and the display, the controller configured to: determine an identity of each of one or more of the food items; determine a target location for each of one or more of the food items based on the identity each of one or more of the food items and an environmental condition of the interior; generate an augmented reality representation including a photographic representation of the interior with a digital representation of the food item overlaying a digital representation of a position of the food item in the interior, wherein the image data includes the augmented reality representation; and communicate a first instruction to present the augmented reality representation with an indication of each target location at the display.
 12. The inventory management system according to claim 11, wherein presenting the augmented reality representation includes displaying food item overlays corresponding to the plurality of food items.
 13. The inventory management system according to claim 11, further comprising: a lighting system in communication with the controller and configured to project a pattern of light in the interior, wherein the controller is further configured to control the lighting system to indicate the target location via the pattern of a light.
 14. The inventory management system according to claim 13, wherein the controller is further configured to control the lighting system to display a color spectrum in the interior indicating the environmental condition.
 15. The inventory management system according to claim 11, wherein the image data includes an overlay corresponding to a property of the food item.
 16. The inventory management system according to claim 15, wherein the property is a fill level and the overlay is a graphical image configured to indicate the fill level of the food item.
 17. The inventory management system according to claim 15, wherein the property is an expiration date and the overlay is a graphical image indicating the expiration date of the food item.
 18. The inventory management system according to claim 11, wherein the controller is further configured to: receive recipe data corresponding to a recipe incorporating one or more of the plurality of food items; communicate a second instruction to present a color indictor indicating positions of the one or more of the plurality of food items at the display.
 19. An inventory management system for an appliance, the inventory management system comprising: a holder disposed in an interior of the appliance and configured to store a food item; an imaging device configured to capture an image of the interior; a display in communication with the imaging device configured to display image data corresponding to the image; and a controller in communication with the imager and the display, the controller configured to: determine a position of the food item based on the image; determine a target location for the food item based on an environmental condition of the interior; compare the position to the target location; modify the image data to indicate the target location based on the comparison of the position to the target location; and communicate an instruction to present the image data at the display.
 20. The inventory management system according to claim 19, wherein the image data includes an augmented reality representation including a photographic representation of the interior with a digital representation of the food item overlaying a digital representation of a position of the food item in the interior, wherein the photographic representation of the interior includes a color indicator indicating the comparison of the position to the target location, and wherein the environmental condition is a heat distribution of the interior. 